<html>
<head>
  <title>supervised learning：predicting an output variable from high-dimensional observations</title>
  <basefont face="微软雅黑" size="2" />
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="exporter-version" content="Evernote Windows/307027 (zh-CN, DDL); Windows/6.1.0 (Win32);"/>
  <style>
    body, td {
      font-family: 微软雅黑;
      font-size: 10pt;
    }
  </style>
</head>
<body>
<a name="878"/>
<h1>supervised learning：predicting an output variable from high-dimensional observations</h1>

<div>
<span><div><div><div><div><div>目录：</div><ul><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/linear_model.html">1.1. Generalized Linear Models</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/lda_qda.html">1.2. Linear and Quadratic Discriminant Analysis</a></li><li style="background-color: white;"><a href="http://scikit-learn.org/stable/modules/kernel_ridge.html">1.3. Kernel ridge regression</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/svm.html">1.4. Support Vector Machines</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/sgd.html">1.5. Stochastic Gradient Descent</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/neighbors.html">1.6. Nearest Neighbors</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/gaussian_process.html">1.7. Gaussian Processes</a></li><li style="background-color: white;"><a href="http://scikit-learn.org/stable/modules/cross_decomposition.html">1.8. Cross decomposition</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/naive_bayes.html">1.9. Naive Bayes</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/tree.html">1.10. Decision Trees</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/ensemble.html">1.11. Ensemble methods</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/multiclass.html">1.12. Multiclass and multilabel algorithms</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/feature_selection.html">1.13. Feature selection</a></li><li style="cursor: pointer; background-color: white;"><a href="http://scikit-learn.org/stable/modules/label_propagation.html">1.14. Semi-Supervised</a></li><li style="background-color: white;"><a href="http://scikit-learn.org/stable/modules/isotonic.html">1.15. Isotonic regression</a></li><li style="background-color: white;"><a href="http://scikit-learn.org/stable/modules/calibration.html">1.16. Probability calibration</a></li><li style="cursor: pointer; background-color: rgb(208, 208, 208);"><a href="http://scikit-learn.org/stable/modules/neural_networks_supervised.html">1.17. Neural network models (supervised)</a></li></ul><div><br/></div><div><img src="supervised learning：predicting an output vari_files/Image.png" type="image/png" data-filename="Image.png" style="font-weight: bold; font-size: 12pt; color: rgb(173, 0, 0);"/></div><div><font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">The problem solved in supervised learning</span></font></div><div style="margin-top: 1em; margin-bottom: 1em;"><a href="http://scikit-learn.org/stable/supervised_learning.html#supervised-learning">Supervised learning</a> consists in learning the link between two datasets: the observed data X and an external variable y that we are trying to predict, usually called “target” or “labels”. Most often, y is a 1D array of length n_samples.</div><div>All supervised <a href="https://en.wikipedia.org/wiki/Estimator">estimators</a> in scikit-learn implement a fit(X, y) method to fit the model and a predict(X) method that, given unlabeled observations X, returns the predicted labels y.</div><div><br/></div><div><font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">Vocabulary: classification and regression</span></font></div><div>If the prediction task is to classify the observations in a set of finite labels, in other words to “name” the objects observed, the task is said to be a <span style="font-weight: bold;">classification</span> task. On the other hand, if the goal is to predict a continuous target variable, it is said to be a <span style="font-weight: bold;">regression</span> task.</div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Nearest neighbor and the curse of dimensionality</span></h2><h3>k-Nearest neighbors classifier</h3><h3>The curse of dimensionality</h3><div>For an estimator to be effective, you need the distance between neighboring points to be less than some value <img src="supervised learning：predicting an output vari_files/Image [1].png" type="image/png" data-filename="Image.png"/>, which depends on the problem. In one dimension, this requires on average <img src="supervised learning：predicting an output vari_files/Image [2].png" type="image/png" data-filename="Image.png"/> points. In the context of the above <img src="supervised learning：predicting an output vari_files/Image [3].png" type="image/png" data-filename="Image.png"/>-NN example, if the data is described by just one feature with values ranging from 0 to 1 and with <img src="supervised learning：predicting an output vari_files/Image [4].png" type="image/png" data-filename="Image.png"/> training observations, then new data will be no further away than <img src="supervised learning：predicting an output vari_files/Image [5].png" type="image/png" data-filename="Image.png"/>. Therefore, the nearest neighbor decision rule will be efficient as soon as <img src="supervised learning：predicting an output vari_files/Image [6].png" type="image/png" data-filename="Image.png"/> is small compared to the scale of between-class feature variations.</div><div style="margin-top: 1em; margin-bottom: 1em;">If the number of features is <img src="supervised learning：predicting an output vari_files/Image [7].png" type="image/png" data-filename="Image.png"/>, you now require <img src="supervised learning：predicting an output vari_files/Image [8].png" type="image/png" data-filename="Image.png"/> points. Let’s say that we require 10 points in one dimension: now <img src="supervised learning：predicting an output vari_files/Image [9].png" type="image/png" data-filename="Image.png"/> points are required in <img src="supervised learning：predicting an output vari_files/Image [10].png" type="image/png" data-filename="Image.png"/> dimensions to pave the <img src="supervised learning：predicting an output vari_files/Image [11].png" type="image/png" data-filename="Image.png"/> space. As <img src="supervised learning：predicting an output vari_files/Image [12].png" type="image/png" data-filename="Image.png"/> becomes large, the number of training points required for a good estimator grows exponentially.</div><div style="margin-top: 1em; margin-bottom: 1em;">For example, if each point is just a single number (8 bytes), then an effective <img src="supervised learning：predicting an output vari_files/Image [13].png" type="image/png" data-filename="Image.png"/>-NN estimator in a paltry <img src="supervised learning：predicting an output vari_files/Image [14].png" type="image/png" data-filename="Image.png"/> dimensions would require more training data than the current estimated size of the entire internet (±1000 Exabytes or so).</div><div>This is called the <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">curse of dimensionality</a> and is a core problem that machine learning addresses.</div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Linear model: from regression to sparsity</span></h2><div>Diabetes dataset</div><div style="margin-top: 1em; margin-bottom: 1em;">The diabetes dataset consists of 10 physiological variables (age, sex, weight, blood pressure) measure on 442 patients, and an indication of disease progression after one year:</div><div style="position: relative;"><div><span style="cursor: pointer; position: absolute; top: 0px; right: 0px; border: 0.916667px solid rgb(221, 221, 221); border-top-left-radius: 0px; border-top-right-radius: 3px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; color: rgb(221, 221, 221); font-family: monospace;" title="Hide the prompts and output">&gt;&gt;&gt;</span></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; diabetes = datasets.load_diabetes()</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; diabetes_X_train = diabetes.data[:-20]</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; diabetes_X_test  = diabetes.data[-20:]</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; diabetes_y_train = diabetes.target[:-20]</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; diabetes_y_test  = diabetes.target[-20:]</span></div></div></div><div>The task at hand is to predict disease progression from physiological variables.</div><div><ul><li><span style="line-height: 1.45;">Linear regression:</span><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression" style="line-height: 1.45;" title="sklearn.linear_model.LinearRegression">LinearRegression</a><span style="line-height: 1.45;">, in its simplest form, fits a linear model to the data set by adjusting a set of parameters in order to make the sum of the squared residuals of the model as small as possible.</span></li></ul></div></div><h3>Shrinkage</h3><div>If there are few data points per dimension, noise in the observations induces high variance.<span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">A solution in high-dimensional statistical learning is to</span> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-style: italic; font-weight: bold;">shrink</span> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">the regression coefficients to zero: any two randomly chosen set of observations are likely to be uncorrelated. This is called</span> <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge" style="font-size: 12pt; color: rgb(28, 51, 135); font-weight: bold;" title="sklearn.linear_model.Ridge">Ridge</a> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">regression</span></div></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><h1 style="text-align: center;"><span style="font-size: 24pt; color: rgb(28, 51, 135); font-weight: bold;">Generalized Linear Models</span></h1><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Ordinary Least Squares</span></h2><div>LinearRegression fits a linear model with coefficients  <img src="supervised learning：predicting an output vari_files/Image [15].png" type="image/png" data-filename="Image.png"/> to minimize the residual sum of squares betweent the observed responses in the dataset and the responses predicted by the linear approximation.</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [16].png" type="image/png" data-filename="Image.png"/></div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [17].png" type="image/png" data-filename="Image.png" style="width: 320.0px; height: 240.0px;"/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Ridge Regression</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge" title="sklearn.linear_model.Ridge">Ridge</a> regression addresses some of the problems of <a href="http://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares">Ordinary Least Squares</a> by imposing a penalty on the size of coefficients. The ridge coefficients minimize a penalized residual sum of squares,</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><img src="supervised learning：predicting an output vari_files/Image [18].png" type="image/png" data-filename="Image.png"/></div><div style="margin-top: 1em; margin-bottom: 1em;">Here, <img src="supervised learning：predicting an output vari_files/Image [19].png" type="image/png" data-filename="Image.png"/> is a complexity parameter that controls the amount of shrinkage: the larger the value of <img src="supervised learning：predicting an output vari_files/Image [20].png" type="image/png" data-filename="Image.png"/>, the greater the amount of shrinkage and thus the coefficients become more robust to collinearity.</div><div style="text-align: center;"><a href="http://scikit-learn.org/stable/auto_examples/linear_model/plot_ridge_path.html"><img src="supervised learning：predicting an output vari_files/Image [21].png" type="image/png" data-filename="Image.png" style="width: 320.0px; height: 240.0px;"/></a></div><h3>Ridge Complexity</h3><div>This method has the same order of complexity than an <a href="http://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares">Ordinary Least Squares</a></div><div><br/></div><h3>Setting the regularization parameter: generalized Cross-Validation</h3><div><font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">RidgeCV</span></font> implements ridge regression with built-in cross-validation of the alpha parameter. The object works in the same way as GridSearchCV except that it defaults to Generalized Cross-Validation (GCV), an efficient form of leave-one-out cross-validation:</div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; from sklearn import linear_model</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; reg = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0])</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; reg.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1])</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">RidgeCV(alphas=[0.1, 1.0, 10.0], cv=None, fit_intercept=True, scoring=None, normalize=False)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; reg.alpha_</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">0.1</span></div></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Lasso</span></h2><div><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">The</span> <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso" style="font-size: 12pt; color: rgb(28, 51, 135); font-weight: bold;" title="sklearn.linear_model.Lasso">Lasso</a> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">is a linear model that estimates sparse coefficients.</span></div><div>Mathematically, it consists of a linear model trained with <img src="supervised learning：predicting an output vari_files/Image [22].png" type="image/png" data-filename="Image.png"/> prior as regularizer. The objective function to minimize is:</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [23].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div>        LASSO回归的特点是在拟合广义线性模型的同时进行变量筛选(Variable Selection)和复杂度调整(Regularization)。 因此，不论目标因变量(dependent/response varaible)是连续的(continuous)，还是二元或者多元离散的(discrete)， 都可以用LASSO回归建模然后预测。 这里的变量筛选是指不把所有的变量都放入模型中进行拟合，而是有选择的把变量放入模型从而得到更好的性能参数。 复杂度调整是指通过一系列参数控制模型的复杂度，从而避免过度拟合(Overfitting)。</div><h3>Setting regularization parameter</h3><div>        scikit-learn exposes objects that set the Lasso alpha parameter by cross-validation: <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoCV.html#sklearn.linear_model.LassoCV" title="sklearn.linear_model.LassoCV">LassoCV</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsCV.html#sklearn.linear_model.LassoLarsCV" title="sklearn.linear_model.LassoLarsCV">LassoLarsCV</a>. <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsCV.html#sklearn.linear_model.LassoLarsCV" title="sklearn.linear_model.LassoLarsCV">LassoLarsCV</a> is based on the <a href="http://scikit-learn.org/stable/modules/linear_model.html#least-angle-regression">Least Angle Regression</a> algorithm explained below.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Multi-task Lasso</span></h2><div>        The <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.MultiTaskLasso.html#sklearn.linear_model.MultiTaskLasso" title="sklearn.linear_model.MultiTaskLasso">MultiTaskLasso</a> is a linear model that estimates sparse coefficients for multiple regression problems jointly: y is a 2D array, of shape (n_samples, n_tasks). <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">The constraint is that the selected features are the same for all the regression problems, also called tasks</span></font>.</div><div>Mathematically, it consists of a linear model trained with a mixed <img src="supervised learning：predicting an output vari_files/Image [24].png" type="image/png" data-filename="Image.png"/> <img src="supervised learning：predicting an output vari_files/Image [25].png" type="image/png" data-filename="Image.png"/> prior as regularizer. The objective function to minimize is:</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><img src="supervised learning：predicting an output vari_files/Image [26].png" type="image/png" data-filename="Image.png"/></div><div style="margin-top: 1em; margin-bottom: 1em;">where <img src="supervised learning：predicting an output vari_files/Image [27].png" type="image/png" data-filename="Image.png"/> indicates the Frobenius norm:</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><img src="supervised learning：predicting an output vari_files/Image [28].png" type="image/png" data-filename="Image.png"/></div><div style="margin-top: 1em; margin-bottom: 1em;">and <img src="supervised learning：predicting an output vari_files/Image [29].png" type="image/png" data-filename="Image.png"/> <img src="supervised learning：predicting an output vari_files/Image [30].png" type="image/png" data-filename="Image.png"/> reads:</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [31].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Elastic Net</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html#sklearn.linear_model.ElasticNet" style="font-size: 12pt; color: rgb(28, 51, 135); font-weight: bold;">ElasticNet</a> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">is a linear regression model trained with L1 and L2 prior as regularizer</span>. This combination allows for learning a sparse model where few of the weights are non-zero like <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso" title="sklearn.linear_model.Lasso">Lasso</a>, while still maintaining the regularization properties of <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge" title="sklearn.linear_model.Ridge">Ridge</a>. We control the convex combination of L1 and L2 using the l1_ratio parameter.</div><div style="margin-top: 1em; margin-bottom: 1em;">Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.</div><div style="margin-top: 1em; margin-bottom: 1em;">A practical advantage of trading-off between Lasso and Ridge is it allows Elastic-Net to inherit some of Ridge’s stability under rotation.</div><div style="margin-top: 1em; margin-bottom: 1em;">The objective function to minimize is in this case</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [32].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Multi-task Elastic Net</span></h2><div>Mathematically, it consists of a linear model trained with a mixed <img src="supervised learning：predicting an output vari_files/Image [33].png" type="image/png" data-filename="Image.png"/> <img src="supervised learning：predicting an output vari_files/Image [34].png" type="image/png" data-filename="Image.png"/> prior and <img src="supervised learning：predicting an output vari_files/Image [35].png" type="image/png" data-filename="Image.png"/> prior as regularizer. The objective function to minimize is:</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><img src="supervised learning：predicting an output vari_files/Image [36].png" type="image/png" data-filename="Image.png"/></div><div>The implementation in the class <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.MultiTaskElasticNet.html#sklearn.linear_model.MultiTaskElasticNet" title="sklearn.linear_model.MultiTaskElasticNet">MultiTaskElasticNet</a> uses coordinate descent as the algorithm to fit the coefficients.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Least Angle Regression</span></h2><div>        Least-angle regression (LARS) is a regression algorithm <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">for high-dimensional data</span></font>, developed by Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani. LARS is similar to forward stepwise regression. <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">At each step, it finds the predictor most correlated with the response.</span></font> When there are multiple predictors having equal correlation, instead of continuing along the same predictor, it proceeds in a direction equiangular between the predictors.</div><div><br/></div><div>The LARS model can be used using estimator <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lars.html#sklearn.linear_model.Lars" title="sklearn.linear_model.Lars">Lars</a>, or its low-level implementation <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.lars_path.html#sklearn.linear_model.lars_path" title="sklearn.linear_model.lars_path">lars_path</a></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">LARS Lasso</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLars.html#sklearn.linear_model.LassoLars" title="sklearn.linear_model.LassoLars">LassoLars</a> is a lasso model implemented using the LARS algorithm, and unlike the implementation based on coordinate_descent, this yields the exact solution, which is piecewise linear as a function of the <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">norm of its coefficients</span></font>.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Orthogonal Matching Pursuit (OMP)</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.OrthogonalMatchingPursuit.html#sklearn.linear_model.OrthogonalMatchingPursuit" title="sklearn.linear_model.OrthogonalMatchingPursuit">        OrthogonalMatchingPursuit</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.orthogonal_mp.html#sklearn.linear_model.orthogonal_mp" title="sklearn.linear_model.orthogonal_mp">orthogonal_mp</a> implements the OMP algorithm for approximating the fit of a linear model with constraints imposed on the number of non-zero coefficients (ie. the L <span style="vertical-align: sub; font-size: smaller;">0</span> pseudo-norm).</div><div>        OMP is based on a greedy algorithm that includes at each step the atom most highly correlated with the current residual. It is similar to the simpler matching pursuit (MP) method, but better in that at each iteration, the residual is recomputed using an orthogonal projection on the space of the previously chosen dictionary elements.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Bayesian Regression</span><span style="line-height: 1.45;">  </span><span style="line-height: 1.45;"> </span></h2><h3>Bayesian Ridge Regression</h3><div>   Bayesian regression techniques can be used to include regularization parameters in the estimation procedure: the regularization parameter is not set in a hard sense but tuned to the data at hand</div><div><ul><li>reg = linear_model.BayesianRidge()</li></ul></div><h3>Automatic Relevance Determination - ARD</h3><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html#sklearn.linear_model.ARDRegression" title="sklearn.linear_model.ARDRegression">        ARDRegression</a> is very similar to <a href="http://scikit-learn.org/stable/modules/linear_model.html#id10">Bayesian Ridge Regression</a>, but can lead to sparser weights <img src="supervised learning：predicting an output vari_files/Image [37].png" type="image/png" data-filename="Image.png"/> <a href="http://scikit-learn.org/stable/modules/linear_model.html#id15">[1]</a> <a href="http://scikit-learn.org/stable/modules/linear_model.html#id16">[2]</a>. <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html#sklearn.linear_model.ARDRegression" title="sklearn.linear_model.ARDRegression">ARDRegression</a> poses a different prior over <img src="supervised learning：predicting an output vari_files/Image [38].png" type="image/png" data-filename="Image.png"/>, by dropping the assumption of the Gaussian being spherical.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Logistic regression</span></h2><div><br/></div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Stochastic Gradient Descent - SGD</span></h2><div>It is particularly useful when the number of samples (and the number of features) is very large. The partial_fit method allows only/out-of-core learning.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Perceptron</span></h2><div>The <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html#sklearn.linear_model.Perceptron" title="sklearn.linear_model.Perceptron">Perceptron</a> is another simple algorithm suitable for large scale learning. By default:</div><blockquote><ul><li>It does not require a learning rate.</li><li>It is not regularized (penalized).</li><li>It updates its model only on mistakes.</li></ul></blockquote><div>The last characteristic implies that the Perceptron is slightly faster to train than SGD with the hinge loss and that the resulting models are sparser.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Passive Aggressive Algorithms</span></h2><div>The passive-aggressive algorithms are a family of algorithms for large-scale learning. They are similar to the Perceptron in that they do not require a learning rate. However, contrary to the Perceptron, they include a regularization parameter C.</div><div>For classification, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.PassiveAggressiveClassifier.html#sklearn.linear_model.PassiveAggressiveClassifier" title="sklearn.linear_model.PassiveAggressiveClassifier">PassiveAggressiveClassifier</a> can be used with loss='hinge' (PA-I) or loss='squared_hinge' (PA-II). For regression, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.PassiveAggressiveRegressor.html#sklearn.linear_model.PassiveAggressiveRegressor" title="sklearn.linear_model.PassiveAggressiveRegressor">PassiveAggressiveRegressor</a> can be used with loss='epsilon_insensitive' (PA-I) or loss='squared_epsilon_insensitive' (PA-II).</div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Robustness regression: outliers and modeling errors</span></h2><div>Robust regression is interested in fitting a regression model in the presence of <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">corrupt data</span></font>: either outliers, or error in the model.</div><div><br/></div><h3>Different scenario and useful concepts</h3><h3>RANSAC: RANdom SAmple Consensus</h3><h3>Theil-Sen estimator: generalized-median-based estimator</h3><h3>Huber Regression</h3><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Polynomial regression: extending linear models with basis functions</span></h2><div>        One common pattern within machine learning is to use linear models trained on nonlinear functions of the data. This approach maintains the generally fast performance of linear methods, while allowing them to fit a much wider range of data.</div><div><br/></div><div><br/></div><h1 style="text-align: center;"><span style="font-size: 24pt; color: rgb(28, 51, 135);">Linear and Quadratic Discriminant Analysis</span></h1><div style="text-align: center;"><br/></div><div>Linear Discriminant Analysis (<a href="http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis" title="sklearn.discriminant_analysis.LinearDiscriminantAnalysis">discriminant_analysis.LinearDiscriminantAnalysis</a>) and Quadratic Discriminant Analysis (<a href="http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis.html#sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis" title="sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis">discriminant_analysis.QuadraticDiscriminantAnalysis</a>) are two classic classifiers, with, as their names suggest, a linear and a quadratic decision surface, respectively.</div><div style="margin-top: 1em; margin-bottom: 1em;">These classifiers are attractive because they have closed-form solutions that can be easily computed, are inherently multiclass, have proven to work well in practice and have no hyperparameters to tune.</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><a href="http://scikit-learn.org/stable/auto_examples/classification/plot_lda_qda.html"><img src="supervised learning：predicting an output vari_files/Image [39].png" type="image/png" data-filename="Image.png" style="width: 512.0px; height: 384.0px;"/></a></div><div>The plot shows decision boundaries for Linear Discriminant Analysis and Quadratic Discriminant Analysis. The bottom row demonstrates that Linear Discriminant Analysis can only learn linear boundaries, while Quadratic Discriminant Analysis can learn quadratic boundaries and is therefore more flexible.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Dimensionality reduction using Linear Discriminant Analysis</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis" title="sklearn.discriminant_analysis.LinearDiscriminantAnalysis">        discriminant_analysis.LinearDiscriminantAnalysis</a> can be used to <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">perform supervised dimensionality reduction</span></font>, by projecting the input data to a linear subspace consisting of the directions which maximize the separation between classes (in a precise sense discussed in the mathematics section below).</div><div><br/></div><div><font style="font-size: 24pt;"><span style="font-size: 24pt; color: rgb(50, 135, 18); font-weight: bold;">理解：类似PCA</span></font></div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Mathematical formulation of the LDA and QDA classifiers</span></h2><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Mathematical formulation of LDA dimensionality reduction</span></h2><div>        To understand the use of LDA in dimensionality reduction, it is useful to start with a geometric reformulation of the LDA classification rule explained above. We write <img src="supervised learning：predicting an output vari_files/Image [40].png" type="image/png" data-filename="Image.png"/> for the total number of target classes. Since in LDA we assume that all classes have the same estimated covariance <img src="supervised learning：predicting an output vari_files/Image [41].png" type="image/png" data-filename="Image.png"/>, we can rescale the data so that this covariance is the identity:</div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><img src="supervised learning：predicting an output vari_files/Image [42].png" type="image/png" data-filename="Image.png"/></div><div style="margin-top: 1em; margin-bottom: 1em;">        Then one can show that to classify a data point after scaling is equivalent to finding the estimated class mean <img src="supervised learning：predicting an output vari_files/Image [43].png" type="image/png" data-filename="Image.png"/> which is closest to the data point in the Euclidean distance. But this can be done just as well after projecting on the <img src="supervised learning：predicting an output vari_files/Image [44].png" type="image/png" data-filename="Image.png"/> affine subspace <img src="supervised learning：predicting an output vari_files/Image [45].png" type="image/png" data-filename="Image.png"/> generated by all the <img src="supervised learning：predicting an output vari_files/Image [46].png" type="image/png" data-filename="Image.png"/> for all classes. This shows that, implicit in the LDA classifier, there is a dimensionality reduction by linear projection onto a <img src="supervised learning：predicting an output vari_files/Image [47].png" type="image/png" data-filename="Image.png"/> dimensional space.</div><div>        We can reduce the dimension even more, to a chosen <img src="supervised learning：predicting an output vari_files/Image [48].png" type="image/png" data-filename="Image.png"/>, by projecting onto the linear subspace <img src="supervised learning：predicting an output vari_files/Image [49].png" type="image/png" data-filename="Image.png"/> which maximize the variance of the <img src="supervised learning：predicting an output vari_files/Image [50].png" type="image/png" data-filename="Image.png"/> after projection (in effect, we are doing a form of PCA for the transformed class means <img src="supervised learning：predicting an output vari_files/Image [51].png" type="image/png" data-filename="Image.png"/>). This <img src="supervised learning：predicting an output vari_files/Image [52].png" type="image/png" data-filename="Image.png"/> corresponds to the n_components parameter used in the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis.transform" title="sklearn.discriminant_analysis.LinearDiscriminantAnalysis.transform">discriminant_analysis.LinearDiscriminantAnalysis.transform</a> method. See <a href="http://scikit-learn.org/stable/modules/lda_qda.html#id4">[3]</a> for more details.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Shrinkage</span></h2><div>        Shrinkage is a tool to improve estimation of covariance matrices in situations where the number of training samples is small compared to the number of features.</div><div style="text-align: center;"><div><img src="supervised learning：predicting an output vari_files/Image [53].png" type="image/png" data-filename="Image.png" style="width: 480.0px; height: 360.0px;"/></div></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Estimation algorithms</span></h2><div>The <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">default solver is ‘svd’</span></font>. It can perform both classification and transform, and it does not rely on the calculation of the covariance matrix. This can be an advantage in situations where the number of features is large. However, the ‘svd’ solver cannot be used with shrinkage.</div><div style="margin-top: 1em; margin-bottom: 1em;">The <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">‘lsqr’ solver</span></font> is an efficient algorithm that only works for classification. It supports shrinkage.</div><div>The <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">‘eigen’ solve</span></font>r is based on the optimization of the between class scatter to within class scatter ratio. It can be used for both classification and transform, and it supports shrinkage. However, the ‘eigen’ solver needs to compute the covariance matrix, so it might not be suitable for situations with a high number of features.</div><div><br/></div><h1 style="text-align: center;"><span style="font-size: 24pt; color: rgb(28, 51, 135);">Kernel ridge regression</span></h1><div>        Kernel ridge regression (KRR) <a href="http://scikit-learn.org/stable/modules/kernel_ridge.html#m2012">[M2012]</a> combines <a href="http://scikit-learn.org/stable/modules/linear_model.html#ridge-regression">Ridge Regression</a> (linear least squares with l2-norm regularization) with the kernel trick. It thus learns a linear function in the space induced by the respective kernel and the data. For non-linear kernels, this corresponds to a non-linear function in the original space.</div><div>        The form of the model learned by <a href="http://scikit-learn.org/stable/modules/generated/sklearn.kernel_ridge.KernelRidge.html#sklearn.kernel_ridge.KernelRidge" title="sklearn.kernel_ridge.KernelRidge">KernelRidge</a> is identical to support vector regression (SVR).</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [54].png" type="image/png" data-filename="Image.png" width="616"/></div><div><br/></div><div><br/></div><h1 style="text-align: center;"><span style="font-size: 24pt; color: rgb(28, 51, 135);">Support Vector Machines</span></h1><div><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">Support vector machines (SVMs) are a set of supervised learning methods used for</span> <a href="http://scikit-learn.org/stable/modules/svm.html#svm-classification" style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">classification</a><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">,</span> <a href="http://scikit-learn.org/stable/modules/svm.html#svm-regression" style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">regression</a> <span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">and</span> <a href="http://scikit-learn.org/stable/modules/svm.html#svm-outlier-detection" style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">outliers detection</a><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">.</span></div><div><br/></div><div>The advantages of support vector machines are:</div><blockquote><ul><li>Effective in high dimensional spaces.</li><li>Still effective in cases where number of dimensions is greater than the number of samples.</li><li>Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.</li><li>Versatile: different <a href="http://scikit-learn.org/stable/modules/svm.html#svm-kernels">Kernel functions</a> can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.</li></ul></blockquote><div style="margin-top: 1em; margin-bottom: 1em;">The disadvantages of support vector machines include:</div><blockquote><ul><li>If the number of features is much greater than the number of samples, avoid over-fitting in choosing <a href="http://scikit-learn.org/stable/modules/svm.html#svm-kernels">Kernel functions</a> and regularization term is crucial.</li><li>SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation (see <a href="http://scikit-learn.org/stable/modules/svm.html#scores-probabilities">Scores and probabilities</a>, below).</li></ul></blockquote><div>The support vector machines in scikit-learn support both dense (numpy.ndarray and convertible to that by numpy.asarray) and sparse (any scipy.sparse) sample vectors as input. However, to use an SVM to make predictions for sparse data, it must have been fit on such data. For optimal performance, use C-ordered numpy.ndarray (dense) or scipy.sparse.csr_matrix (sparse) with dtype=float64.</div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Classification</span></h2><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" title="sklearn.svm.SVC">        SVC</a>, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVC.html#sklearn.svm.NuSVC" title="sklearn.svm.NuSVC">NuSVC</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC" title="sklearn.svm.LinearSVC">LinearSVC</a> are classes capable of performing multi-class classification on a dataset.</div><div><a href="http://scikit-learn.org/stable/auto_examples/svm/plot_iris.html"><img src="supervised learning：predicting an output vari_files/Image [55].png" type="image/png" data-filename="Image.png"/></a></div><div>        As other classifiers, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" title="sklearn.svm.SVC">SVC</a>, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVC.html#sklearn.svm.NuSVC" title="sklearn.svm.NuSVC">NuSVC</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC" title="sklearn.svm.LinearSVC">LinearSVC</a> take as input two arrays: an array X of size [n_samples, n_features] holding the training samples, and an array y of class labels (strings or integers), size [n_samples]:</div><div><br/></div><h3>Multi-class classification</h3><div><a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" title="sklearn.svm.SVC">        SVC</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVC.html#sklearn.svm.NuSVC" title="sklearn.svm.NuSVC">NuSVC</a> implement the “one-against-one” approach (Knerr et al., 1990) for multi- class classification. If n_class is the number of classes, then <font style="font-size: 12pt;"><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">n_class * (n_class - 1) / 2 classifiers</span></font> are constructed and each one trains data from two classes.</div><div><br/></div><div><font style="font-size: 14pt;"><span style="font-size: 14pt; font-weight: bold; line-height: 1.45;">Scores and probabilities</span></font></div><div><br/></div><div>        The <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" title="sklearn.svm.SVC">SVC</a> method decision_function gives per-class scores for each sample (or a single score per sample in the binary case). When the constructor option probability is set to True, class membership probability estimates (from the methods predict_proba and predict_log_proba) are enabled.</div><div><br/></div><h3>Unbalanced problems</h3><div>        In problems where it is desired to give more importance to certain classes or certain individual samples keywords class_weight and sample_weight can be used.</div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [56].png" type="image/png" data-filename="Image.png" style="width: 480.0px; height: 360.0px;"/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Regression</span></h2><div>        There are three different implementations of Support Vector Regression: <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR" title="sklearn.svm.SVR">SVR</a>, <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVR.html#sklearn.svm.NuSVR" title="sklearn.svm.NuSVR">NuSVR</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR" title="sklearn.svm.LinearSVR">LinearSVR</a>. <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR" title="sklearn.svm.LinearSVR">LinearSVR</a> provides a faster implementation than <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR" title="sklearn.svm.SVR">SVR</a> but only considers linear kernels, while <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVR.html#sklearn.svm.NuSVR" title="sklearn.svm.NuSVR">NuSVR</a> implements a slightly different formulation than <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR" title="sklearn.svm.SVR">SVR</a> and <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVR.html#sklearn.svm.LinearSVR" title="sklearn.svm.LinearSVR">LinearSVR</a>. See <a href="http://scikit-learn.org/stable/modules/svm.html#svm-implementation-details">Implementation details</a> for further details.</div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; from sklearn import svm</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; X = [[0, 0], [2, 2]]</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; y = [0.5, 2.5]</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; clf = svm.SVR()</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; clf.fit(X, y)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">SVR(C=1.0, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='auto', kernel='rbf', max_iter=-1, shrinking=True, tol=0.001, verbose=False)</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">&gt;&gt;&gt; clf.predict([[1, 1]])</span></div><div><span style="font-size: 9pt; background-color: rgb(251, 250, 248); color: rgb(51, 51, 51); font-family: Monaco;">array([ 1.5])</span></div></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Density estimation, novelty detection</span></h2></div><div style="text-align: center;"><img src="supervised learning：predicting an output vari_files/Image [57].png" type="image/png" data-filename="Image.png" style="width: 480.0px; height: 360.0px;"/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Complexity</span></h2><div>        Support Vector Machines are powerful tools, but their compute and storage requirements increase rapidly with the number of training vectors. The core of an SVM is a quadratic programming problem (QP), separating support vectors from the rest of the training data. The QP solver used by this <a href="http://www.csie.ntu.edu.tw/~cjlin/libsvm/">libsvm</a>-based implementation scales between <img src="supervised learning：predicting an output vari_files/Image [58].png" type="image/png" data-filename="Image.png"/> and <img src="supervised learning：predicting an output vari_files/Image [59].png" type="image/png" data-filename="Image.png"/> depending on how efficiently the <a href="http://www.csie.ntu.edu.tw/~cjlin/libsvm/">libsvm</a> cache is used in practice (dataset dependent). If the data is very sparse <img src="supervised learning：predicting an output vari_files/Image [60].png" type="image/png" data-filename="Image.png"/> should be replaced by the average number of non-zero features in a sample vector.</div><div>        Also note that for the linear case, the algorithm used in <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC" title="sklearn.svm.LinearSVC">LinearSVC</a> by the <a href="http://www.csie.ntu.edu.tw/~cjlin/liblinear/">liblinear</a> implementation is much more efficient than its <a href="http://www.csie.ntu.edu.tw/~cjlin/libsvm/">libsvm</a>-based <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" title="sklearn.svm.SVC">SVC</a> counterpart and can scale almost linearly to millions of samples and/or features.</div><div><font style="font-size: 24pt;"><span style="font-size: 24pt; color: rgb(50, 135, 18); font-weight: bold;">理解：这一段是讲svm的复杂度</span></font></div><div><br/></div><h2><br/></h2><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Tips on Practical Use</span></h2><div><ul><li><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">Avoiding data copy</span></li><li><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">Kernel cache size</span></li><li><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">Setting C：<span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">C is 1 by default and it’s a reasonable default choice. If you have a lot of noisy observations you should decrease it. It corresponds to regularize more the estimation.</span></span></li><li><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">recommended to scale your data</span></li></ul></div><div><br/></div><div><br/></div><h2><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">Kernel functions</span></h2><div>The <span style="font-style: italic;">kernel function</span> can be any of the following:</div><blockquote><ul><li>linear: <img src="supervised learning：predicting an output vari_files/Image [61].png" type="image/png" data-filename="Image.png"/>.</li><li>polynomial: <img src="supervised learning：predicting an output vari_files/Image [62].png" type="image/png" data-filename="Image.png"/>. <img src="supervised learning：predicting an output vari_files/Image [63].png" type="image/png" data-filename="Image.png"/> is specified by keyword degree, <img src="supervised learning：predicting an output vari_files/Image [64].png" type="image/png" data-filename="Image.png"/> by coef0.</li><li>rbf: <img src="supervised learning：predicting an output vari_files/Image [65].png" type="image/png" data-filename="Image.png"/>. <img src="supervised learning：predicting an output vari_files/Image [66].png" type="image/png" data-filename="Image.png"/> is specified by keyword gamma, must be greater than 0.</li><li>sigmoid (<img src="supervised learning：predicting an output vari_files/Image [67].png" type="image/png" data-filename="Image.png"/>), where <img src="supervised learning：predicting an output vari_files/Image [68].png" type="image/png" data-filename="Image.png"/> is specified by coef0.</li></ul></blockquote><div style="margin-top: 1em; margin-bottom: 1em;">Different kernels are specified by keyword kernel at initialization:</div><div><span style="cursor: pointer; position: absolute; top: 0px; right: 0px; border: 0.916667px solid rgb(221, 221, 221); border-top-left-radius: 0px; border-top-right-radius: 3px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; color: rgb(221, 221, 221); font-family: monospace;" title="Hide the prompts and output">&gt;&gt;&gt;</span></div><div style="box-sizing: border-box; padding: 8px; font-size: 12px; border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.14902);"><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; linear_svc = svm.SVC(kernel='linear')</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; linear_svc.kernel'linear'</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; rbf_svc = svm.SVC(kernel='rbf')</span></div><div><span style="background-color: rgb(251, 250, 248); font-size: 12px; color: rgb(51, 51, 51); font-family: Monaco, Menlo, Consolas, 'Courier New', monospace;">&gt;&gt;&gt; rbf_svc.kernel'rbf'</span></div></div><div><br/></div><h3>Custom Kernels</h3><div>You can define your own kernels by either giving the kernel as a python function or by precomputing the Gram matrix.</div><div style="margin-top: 1em; margin-bottom: 1em;">Classifiers with custom kernels behave the same way as any other classifiers, except that:</div><blockquote><ul><li>Field support_vectors_ is now empty, only indices of support vectors are stored in support_</li><li>A reference (and not a copy) of the first argument in the fit() method is stored for future reference. If that array changes between the use of fit() and predict() you will have unexpected results.</li></ul></blockquote><div><br/></div><h2><span style="color: rgb(28, 51, 135); font-size: 18pt; font-weight: normal;">Mathematical formulation</span></h2><h2><span style="color: rgb(28, 51, 135); font-size: 18pt;">Implementation details</span></h2><div><div><br/></div><table style="border-collapse: collapse; min-width: 100%;"><colgroup><col style="width: 311px;"></col><col style="width: 308px;"></col></colgroup><tbody><tr><td style="border: 1px solid rgb(204, 204, 204); width: 311px; padding: 8px;"><h3>SVC</h3></td><td style="border: 1px solid rgb(204, 204, 204); width: 308px; padding: 8px;"><div><br/></div></td></tr><tr><td style="border: 1px solid rgb(204, 204, 204); width: 311px; padding: 8px;"><h3>NuSVC</h3></td><td style="border: 1px solid rgb(204, 204, 204); width: 308px; padding: 8px;"><div><br/></div></td></tr><tr><td style="border: 1px solid rgb(204, 204, 204); width: 311px; padding: 8px;"><h3>SVR</h3></td><td style="border: 1px solid rgb(204, 204, 204); width: 308px; padding: 8px;"><div><br/></div></td></tr></tbody></table><div><br/></div></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div></div><div><br/></div></span>
</div></body></html> 